Memory Optimization Techniques in Neural Networks: A Review

Deep neural networks have been continuously evolving towards larger and more complex models to solve challenging problems in the field of AI. The primary bottleneck that restricts new network architectures is memory consumption. Running or training DNNs heavily relies on hardware (CPUs, GPUs, FPGA) which are either inadequate terms hard-to-extend. This would further make it difficult scale. In this paper, we review some latest footprint reduction techniques enable faster low model complexity. Additionally, improves accuracy by increasing batch size developing wider deeper with same set resources. paper emphasizes optimization methods specific CNN RNN training.